Note: Andrew Morton is the Lead Maintainer for the Linux public production kernel. His work is supported by the Open Source Development Labs (OSDL). Morton works with Linux creator Linus Torvalds, the kernel subsystem maintainers, Linux distribution companies, hardware vendors, and other interested parties to ensure that the public production kernel meets their needs. He is the final arbitrator on determining what code is accepted into the Linux production kernel.

Andrew Morton spoke recently at the Software Developers Forum in Palo Alto. This is the text of his presentation.

******************************

Andrew Morton
May 19, 2005
Palo Alto, CA

My aim is to help people understand where all of this open source software comes from and why people would do such a strange thing as to develop software for free. And it is strange – people who I believe are some of the best programmers in the world choose to work on free software.

Free software is very much a take it or leave it kind of thing. We tend to not try to convince people to use it – if it's useful to you then go ahead, but please do honor the license, the General Public License (GPL).

If open source software is not useful to you, then shrug and use something else. It's just another tool available to you when preparing those products which provide value to end users.

We, the Linux developers, don't have a direct stake in whether or not you choose to "purchase" our product, which means that I'm often put in the strange position of giving talks which are designed to benefit the audience, rather than the speaker.

This talk is a bit about free software economics, but mainly it's about the Linux kernel development processes: who the developers are, why they do it, how they do it, how we arrive at decisions about kernel features and how the kernel finds its way into end users' hands.

My comments may be generalized onto other open source projects, but the Linux kernel has several unique aspects. It's highly modular, it's well suited to distributed development across location and time, and it's GCC is different.

1 – Most successful/important free software is "system software": software which other software builds upon to provide a service. This includes the kernel, low-level software libraries, web and other servers, databases, mail servers, and many system tools, etc. This is no coincidence.

2 – The normal processes of market competition do not work with system software, due to high substitution costs. End consumers need to retrain administrative staff and users, they need to obtain new versions of their applications to run on the new OS (or go to a different product if not available), and they need to redevelop applications if they use in-house applications, but still support the old system software. In all cases, they can only run their software on the hardware platforms for which their new vendor has chosen to make the system software available.

3 – Independent Software Vendors: ISV's need to develop, maintain and support their product on various versions of 5, 10 or more different vendors' operating systems. That's a lot of cost which doesn't bring in a single new sale and which inevitably impacts the quality of the overall product. ISV's would much prefer that all the machines in the world be running the same operating system so they only need to develop, maintain and support a single version. So of course what happens is that they develop for the most popular OS amongst their target customers, thus starving off minor players and causing a near insurmountable barrier for new operating system entrants.

4 – Hardware vendors: These vendors need to ensure that their hardware works well with operating system software which is in the field, sometimes going back over 10 years' worth of releases. Often hardware vendors develop and maintain the device drivers for the hardware, which they manufacture, which puts them in same boat as ISV's. The end result is that hardware vendors only support the most popular operating system software, while minor or new entrants have to take on the cost of supporting that hardware themselves – sometimes without adequate hardware documentation.

5 – As a consequence of these high substitution costs, all players tend to gravitate toward a single suite of underlying system software. Which is great for the providers of that software, but obviously places that provider in a monopolistic position, and leaves the ISV users of that software with a single source for a very vital component, often from a competitor.

6 – To get around this fundamental single-provider problem, the industry is doing an amazing thing for which it is hard to see parallels in other fields. Many IT companies are congealing around a suite of Free software, which nobody owns – or if you like, which everybody owns.

7 – This allows many competing industry players to use the same basic set of system software, but without relinquishing control to the provider of that software.

8 - These lower-level system software components are, to be frank, 10, 20 or even 30-year-old technologies. Everyone knows how to write a POSIX-compliant operating system or a C library or a window manager. There is no real intellectual value left in those things; there is no particular product differentiation between the different providers. For commercial OS vendors, these low-level infrastructural products are just cost centers and customer hassles.

With open source the IT companies get to pool development costs to provide an industry-standard software platform which minimizes the cost of their customers switching application providers. Yet the IT companies who provide Linux-based hardware and applications can still make any changes to the operating system which they require.

The IT companies retain a sufficient degree of control over the operating system whilst not bearing the full cost of its maintenance. It pushes this whole stack of boring, underlying software off the table so the software entrepreneurs can concentrate their innovation, investment and talent resources on the higher-level new software where innovation and product differentiation provide real customer value.

In recent years, we've seen numerous large IT companies becoming interested in Linux. Remember that although the sticker price is zero, Linux does have costs to those companies.

They must employ staff to maintain their Linux-based products, staff to support internal teams, staff to add features, all of which their employer requires. Often these features are fed back into the mainline, upstream, open source product – something that can be viewed as part of the acquisition costs of open source software.

The GPL requires that any changes which an organization makes to the software's source code be made available to all the parties to whom that software is distributed as an end product. It turns out that, for various reasons related to both cost and code quality, this ends up leading companies who base their products on Free software to prefer to merge their changes into the public tree rather than continually maintaining their own out-of-tree changes. And this has two really important benefits

for the entire community of open source users.

1 – It tends to pull new software features into the main public codebase, thus minimizing fragmentation. All users of the software see the same industry standard feature set and we don't have lots of organizations running off in different directions adding incompatible changes to Linux.

2 – The GPL provides the open source project with its workforce – we have lots of developers in lots of companies working, out of their employer's self interest, to maintain and add capabilities to the public tree.

If all of this software was made available under a different license – one which permitted the proprietary retention of such changes – the thing called "Linux" would become fragmented and development of the core codebase would dwindle. Companies would run off in different directions adding value for their own competitive advantage and the overall product would fragment. This was the process that happened with Unix.

However, having said all this and contrary to what one hears, GPL'ed software is not "free" in dollar terms for those who integrate it into other products. Any company can use Linux, but it turns out that they have to pay for it. It's not a dollar payment – it is a "payment in kind" and consists of software enhancements and the use of engineers' time.

Overall, the GPL is the key thing which has kept this system running successfully, and years of experience tell us that these arrangements work really well. They have gotten us to where we are today.

When people consider open source software, there is frequently incredulity as to the motivation and resourcing: This software cannot grow on trees, so where does it come from? Why do people work on it? What assurances do we have that the developers will continue to maintain the software?

The stereotype that falls out of these questions is of a male computer geek spending his spare time writing code purely for the love of his craft. And certainly such people were a significant force up to five years ago. But nowadays most core kernel developers are full-time employed professionals, and are contributing to the kernel and other public software projects as part of their job description. There are lots of people, lots of professionals involved in the Linux kernel project.

There have been more than 60,000 changes in the past 3 and a half years alone, involving approximately 1,200 individuals. However, the reality is that the top 20 contributors provided about half of those changes, so you can see it's a heavily skewed contribution rate.

Look at where these people come from:

- A small number are employed by OSDL, which is an industry consortium supported by many IT companies. Linus Torvalds and I are funded by OSDL, partly because we need to operate and to be seen to be operating as independent agents.

- Many major contributors work for Linux distributors, mainly Red Hat and Suse (now Novell). Some of Red Hat and Suse's major customers are large IT corporations, so these companies are effectively funding kernel development via a subcontracting arrangement.

- Some kernel contributors are employed by hardware companies that wish to ensure that Linux works well on their latest products. These include very small companies who have a specialized product line.

- We have contributors from smaller companies. For instance, say a company is need of in-house expertise on that software, so they hire engineers to become part of the development team for the open source products which they are using. These developers split their time between contributing to the external open source component and helping their employer fit that open source component into their final product.

- Individual enthusiasts – they actually still exist in large numbers, but their contributions are waning in a relative sense on the large core open source components as full-timer workers overshadow them. The enthusiasts tend to get hired to do their work full-time anyway.

As a person who has worked with and managed software engineers for my whole career, I have noticed that Free software attracts the very best developers – highly skilled, highly experienced and dedicated perfectionists. In my experience these, people tend to be miles ahead of regular software professionals in both the volume and the quality of their work.

Even though we're all paid to do kernel work, the culture is largely unchanged from the earliest days of Linux. We work on it because we love our craft and because it allows us to contribute to our society in the way in which we are most able. Corny but true.

Consequently, I think it's fair to say that generally an individual kernel developers' first allegiance is to the kernel as a whole rather than to his (or, rarely, her) employer. A kernel developer who works for an IT corporation inevitably has one leg in each camp and is able to act as an interpreter or a gateway between the corporate culture and the kernel community. This means that the developers are able to prevent their companies from making mistakes in the way in which they present

themselves to the kernel community and to the wider Open Source community.

I suspect this interface function, and the fact that developers are more aligned to the kernel project as a whole, serves to explain why we see so little conflict between the various interested parties. It's quite surprising really – we have large and very competitive companies who are up to their necks in kernel development, but publicly at least we never ever see any conflict.

Occasionally, a feature will float past which only the originating company has any interest in. But we can arrange for that feature to have no detrimental impact on any of the other users, and as long as it's sensibly implemented then yes, we can merge that feature as a service to its developer.

Other times a feature will float past which is developed by contributors from one company, but multiple companies do have an interest in it and often their requirements are slightly different. In these situations, I or the relevant subsystem maintainer, will act as a mediator. I'll identify all the stakeholders, make sure that they have taken the time to review and test the work, and perhaps modify it. Once everybody is happy with the feature and I'm happy that sufficient discussion has occurred, and I can see that nobody's toes are being trodden on, I'll merge that feature into the mainline kernel.

Everybody understands that this is the process and everybody respects the logic behind it and generally there are no surprises and everything goes smoothly.

– Practically all kernel development is performed via email on public lists in which any interested individual can participate. Most of the core kernel team gathers once per year at a conference in Ottawa – partly for technical discussions but also (of course) for human interaction.

– The kernel project has automatically organized itself into component teams, so we have a 3-layer tree-shaped workflow structure wherein individual members of a team will submit their contributions to a team leader. The team leader then integrates these changes, making sure they are tested, and then submits them into the top-level code repository.

We have probably 50 or more subsystems, each with an identifiable top-level maintainer. The kernel breaks down nicely into that model because the various technologies which constitute the kernel are so decoupled.

In some open source projects, the various team leaders will commit the change themselves. In the kernel project, a single individual has final control over what goes in. Despite that, Linus and I will rarely even look at the code which, say, the scsi maintainer sends. We're interested in the timing of the submission and the testing status, but the internal details are under the control of the subsystem maintainer.

– Code submissions are sent via email and are copied to an open mailing list for interested parties to review.

– The code changes are immediately available on Internet servers and are pulled down and tested by 100's or 1000's of people on a real-time basis.

– The main maintainer (mm) aggregates things – this is the "bleeding edge" of the process, and the subsystem maintainers merge after mm testing.

– There are approximately 2000 mm downloaders. That group is a mix of kernel developers and generous enthusiasts, and they are very important to us.

– If defects are detected, they are usually reported via email to the mailing lists and a dialogue will ensue, usually ending up with a code change – a patch. The originator of the bug report will confirm that the problem was fixed and the code change is committed to the top-level tree. Turnaround time for this entire process is typically 2 hours to 2 days.

This is an extremely effective bug-fixing process and is quite unusual. No company would dream of letting end-users directly enter into discussion of a problem with their core product developers. But we do, and it turns out that it works.

I liken kernel development to coding in a fishbowl – every edit is watched over by hundreds of individuals and is immediately tested by thousands. Any design discussions are read by thousands and are discussed on several Internet sites which are dedicated to this purpose.

People like Linus Torvald and I don't plan the kernel evolution. We don't sit there and think up the roadmap for the next two years, then assign resources to the various new features. That's because we don't have any resources. The resources are all owned by the various corporations who use and contribute to Linux, as well as by the various independent contributors out there. It's those people who own the resources who decide what feature work they'll devote those resources to.

This actually tends to work out quite well – if nobody is prepared to pony up the programmer hours for a particular feature, then it's a fair bet that we didn't want that feature anyway.

A consequence of this is that if your business requires that the kernel supports a particular new feature, the solution is obvious – hire a few programmers, get down and write it, and then send the patch to Andrew Morton. An alternative would be to cut a contract with one of the big Linux distributors, or one of the other smaller shops out there, but that approach seems to be less popular than simply hiring the engineers and getting it done.

Now, if you have developed a new kernel feature then you don't "have" to get that feature merged into the mainline public kernel. You can just develop the feature, add it to your copy of the kernel and ship the result to your customers without attempting to merge the feature back into the public tree. Many companies do this, and some of those manage to comply with the kernel license whilst doing so. Not all, though.

Sometimes it makes technical and commercial sense to merge your new feature into the public tree, and sometimes it does not.

The technical barriers for inclusion are high – we have very exacting quality requirements. And I do always like to be convinced that the submitter of a feature is prepared to support that feature while it is in the public tree – that they won't simply let it rot. It may be the case that you're simply not able to put sufficient resources into the development to bring the code up to mergeable quality. In which case, it probably makes sense to retain the feature as a private thing and to not seek an upstream merge.

Also, the feature may address some obscure requirement or some rare piece of hardware and it simply doesn't make sense for us to redistribute that feature within the public tree – to millions of people who couldn't possibly need it.

But, once we pass these hurdles – if the feature is judged to be generally useful – then you will find that there are significant advantages to merging upstream and not maintaining your own external patchset:

- Others find and fix bugs.
- Others will get the feature working on other architectures.
- Kernel-wide codesweeps fix your code instead of breaking it.
- There are more testers.
- It's more likely that other interested parties will add more features.

How do the kernel development processes differ from those in a regular commercial development shop?

- Planning process is different – push up, not push down
- Bug-fixing process is different
- Less organizational hierarchy
- No managers
- Negative powers, not positive powers
- Less specialization/compartmentalization
- Probably more emphasis on code review
- We don't have schedules.
- Tons of resources – More than 1000 developers, 4500 mailing list subscribers, several thousand daily testers.

– Some users take the kernel.org kernel directly, but they're usually propellorheads.

– Most commonly, a distributor will take a snapshot of the kernel.org kernel when it suits them.

– They will sit on it for a few months, going through a stabilization process.

– They observe the ongoing changes in the public tree as it advances, and backport bug fixes.

– Perform their own QA, raise their own bug fixes, forward-port those into the latest public kernel.

– They go through their own customer beta cycle process

– Eventually they'll release a kernel which consists of the public kernel as of a few months ago, plus a bunch of bug fixes and perhaps features.

– The distributor will then support that kernel for maybe five years.

– They will release service packs, which contain new fixes and new features.

– After a while, the distributor's kernel will be quite different from the original public kernel which they started out with. It might have hundreds and hundreds of patches applied to it, but they never

up-rev the actual kernel version number because that would mean that large applications need to be re-certified by the ISVs, which is expensive.

Hopefully I've demystified a few things, if there was any mystification out there in the first place.

I do view public awareness as part of my job – to help to facilitate the relationship between the kernel development team and those companies which use the kernel and have a need to further develop the kernel.

That effort generally involves explaining kernel culture and processes to technical managers who are less familiar with the open source world.

If any of you find you have concerns with kernel technical issues, or with kernel feature content or with interacting with the kernel development team – or anything like that – I would invite you to contact me directly via email. You can easily use Google to find my contact information.

Usually I can explain things, put the right people in contact with each other, and sometimes I can even prevent people from running off in the wrong direction and wasting lots of time, or duplicating work which has already been done or which is already in progress.

Thank you.

************************

May 19, 2005

Andrew Morton can be reached at OSDL (Open Source Development Labs) in Palo Alto, CA.

Copyright (c) 2005, Andrew Morton. All rights reserved.